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Department of Defense weapon systems are growing more complex and 
expensive and the services are under increasing pressure from Congress to improve the 
cost schedule and performance of their weapons programs. The purpose of this project 
was to investigate and provide an overview of issues within the DoD acquisition system 
and provide information to assist program managers in dealing with some of the 
challenges they face. The goal of this project was to identify Critical Success Factors and 
determine if they could be applied to DoD acquisition programs in order to improve the 
acquisition process and provide program managers with a tool to assist them with 
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I. 


INTRODUCTION 


Over the past half century weapons systems have grown increasingly complex 
and expensive. In 2002 the DoD spent $110 billion to research, develop and acquire a 
wide array of weapons systems.i In 2005, weapons system procurement accounted for 
$144 billion in spending and by 2009 it is estimated that the DoD will spend $185 
billion.^ 



2001 



Program 

Cost 

Program 

Cost 

R''A-22 Raptor aircraft 

$SS.O billion 

Joint Strike Fighter 

$208.3 billion 

DDG-S1 class destroyer ship 


Future Combat Systems 

$127.8 billion 

Virginia class submarine 


Virginia class submarine 

$80.4 billion 

C-17 Globemaster airlift aircraft 

$S1.1 billion 

DDG-S1 class destroyer ship 

$70.4 billion 

R''A-18E/F Super Hornet fighter 
aircraft 

$48.2 billion 

RA-22 Raptor aircraft 

$68.4 billion 

Total 

S20O.8 billion 

Total 

SSS0.0 billion 


Souns: [iiXi ((teSi: (MO (anali^lsand pfesenlslom). 

Figure 1. Total Cost of DoD's Top Five Programs in Fiscal Years 2001 and 2006 (in 
2006 Dollars)(From: Government Accountability Office [GAO], GAO-06- 

257T, 2006) 

Figure 1 shows the DoD's top five programs from 2001 to 2006 and the associated 
cost growth among them. The figure shows significant cost growth among those 
programs and shows that the top five programs have almost doubled in cost from $290.8 
billion to $550.0 billion in the past five years. In 2006 the Government Accountability 
Office made the following observation: 


^ Government Accountability Office, Major Challenges and Program Risks, GAO-03-98, 2003, 50. 

^ Government Accountability Office, Best Practices-Better Support of Weapon System Program 
Managers Needed to Improve Outcomes, GAO-06-110, 2006, 3. 
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current military operations, such as those in Afghanistan and Iraq, 
consume a large share of DOD resources and are causing faster wear on 
existing weapons. Refurbishment or replacement sooner than planned is 
putting further pressure on DOD’s investment accounts. 

At the same time DOD is facing these problems, programs are 
commanding larger budgets. DOD is undertaking new efforts that are 
expected to be the most expensive and complex ever and on which DOD 
is heavily relying to fundamentally transform military operations. And it is 
giving contractors increased program management responsibilities to 
develop requirements, design products, and select major system and 
subsystem contractors. Figure 1 shows that just 5 years ago, the top five 
weapon systems cost about $291 billion combined; today, the top five 
weapon systems cost about $550 billion.3 

With the large monetary value and implications for National defense, it is easy to 
see why Congress and the DoD have placed much emphasis on oversight of weapon 
system acquisition. In developing major weapon systems, program managers have the 
daunting task of balancing many factors that decide the cost, schedule and performance 
of their programs. There are many aspects that can affect weapon system programs, both 
positively and negatively. Many of these factors that affect a program cannot be 
controlled by the program manager. Factors such as political turmoil, changes in budget 
priorities and personnel changes within the DoD can all have disastrous consequences on 
a program. Many of these factors are outside the scope of what a program manager is 
expected to manage. This research will examine the program factors in which the 
program manager has some control. 

Analyzing data from research conducted by the GAO and other relevant sources 
will lead to the identification of key factors that contribute to the success or failure of a 
weapon system programs. 

A. PROJECT OBJECTIVES 

The primary objective of this project is to identify Critical Success Factors (CSF) 
that lead to the success or failure of a program. My intent is to provide the acquisition 


3 Government Accountability Office, DoD Acquisition Outcomes-A Case for Change, GAO-06-257T, 
2006, 3. 
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professional some guidelines to identify success factors in managing complex 
DoD weapon system programs. If a professional is aware of and is able to identify the 
Critical Success Factors that this research identifies, then the acquisition professional will 
be able to employ strategies and gain personal attributes that positively influence the 
success of future programs. By identifying both the technical and educational/experience 
factors in DoD acquisition programs, acquisition professionals will be better able to 
improve the acquisition process in an informed and realistic manner. 

B. RESEARCH QUESTIONS 

1. Primary Research Question 

What are the Critical Success Factors (CSF) that influence the successful 
acquisition of DOD weapon system programs? 

2. Secondary Questions 

• How do we define the success of an acquisition program? 

• What aspects lead to the successful acquisition of a DOD weapon system? 

C. SCOPE OF PROJECT 

This research will: 

• Identify Critical Success Factors that are applicable to DoD acquisition 
programs 

• Provide DoD acquisition professionals an identification of Critical Success 
Factors that have been proven to positively influence the success of DoD 
major acquisition programs. 

D. ORGANIZATION 

Chapter II will discuss the background of program management in the DoD as 
well as the history of weapons system development. This chapter will also examine the 
major changes that have affected the acquisition process. It will also address the parties 
involved in the process that influence the outcome of programs. 

Chapter III will present data on Critical Success Factors that affect a program. 
The research will focus on research published by acquisition professionals and the 
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Government Accountability Office (GAO) as well as past weapons programs and analyze 
factors and events within those programs. This project will identify factors that had a 
major effect on the success or failure of the program. 

Chapter IV analyzes the data presented from Chapter 3 and compares that data to 
identify any correlations between the research that has been conducted the GAO reports 
and any other evidence. 

Chapter V presents the conclusions and the recommendations regarding program 
managers and their programs. 


4 



II. BACKGROUND 


During World War II the acquisition of weapons systems was a relatively simple 
process compared to today. During that period, a need was identified, a contractor chosen 
and a product was produced, usually in very large quantities. The weapons were 
relatively simple to produce by today's standards, and that simplicity resulted in weapon 
systems that could be produced in large quantities by companies that had little experience 
manufacturing weapons. For instance, during WWII roughly 50,000 M4 "Sherman" tanks 
were produced for the US and its allies.^ These tanks were relatively simple and easy to 
produce in large numbers. By contrast, the Ml "Abrams" series tank has only been 
produced in quantities of about 8,800 over a twenty five-year period.^ The Ml tank 
contains computerized fire control systems and other digital systems as well as software 
that have greatly increased the capability of each tank. This capability is why the US 
forces are the best in the world, but with the increased capability comes increased 
complexity. The same increase in complexity we see in tanks can be seen in other 
systems as well, including aircraft. Currently, the F22 and FI8 aircraft are the premier 
fighter aircraft in the US arsenal. These aircraft use computers to control flight control 
surfaces in what is known as fly-by-wire. These systems are very complex compared to 
the planes that were flying even 20 years ago and these complex weapon systems are 
being produced in decreasingly smaller quantities. As Figure 2 illustrates, weapon 
systems are growing in cost and as the cost grows the buying power is reduced resulting 
in fewer items being purchased. 


^ R.P. Hunnicutt. Patton: A History of the American Main Battle Tank. 1984, Presidio Press, 65. 

^ http://www.fas.org/man/dod-101/svs/land/ml.htm . Ml Abrams Tank Production, 01 December 2006. 
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Program 

Initial Initial Latest Latest Percent of unit 

estimate quantity estimate quantity cost increase 

Joint 

Strike 

Fighter 




$189.8 billion 2.866 aircraft $206.3 billion 2,458 aircraft 26.7 

Future 

Combat 

Systems 


f 


$82.6 billion 15 systems $127.5 billion 15 systems 54.4 

F-22A 

Raptor 




$81.1 billion 648 aircraft $65.4 billion 181 aircraft 188.7 

Evolved 

Expendable 

Launch 

Vehicle 


$15.4 billion 181 vehicles $28.0 billion 138 vehicles 137.8 


T 



Space 

Based 

Infrared 
System High 


B 

a 


$4.1 billion 5 satellites $10.2 billion 3 satellites 315.4 

Expeditionary 

Fighting 

Vehicle 



r 

$8.1 billion 1,025 vehicles $11.1 billion 1.025 vehicles 35.9 


SC4f ce; <a*0 enalyste of DOD data, ima^aa eourcad in (h&t re6p»cin<a ordar: JSF F^ograrn Offlcat Rogram Manage*. urrii of acttcn. 
U.& arn^: F* 22a System Reliant Offloa; (Lan>02003 lLS1.ocar>aad Marorx (Tight) Q 2003Ttte Doelng cempany; Locattesd Maidn 
Spaca ^atama Company; Ganami Dynamics Land Systems. 

Figure 2. Examples of DoD Programs with Redueed Buying Power (2006) (From: 

GAO, GAO-06-585T, 2006) 


As Figure 2 illustrates as the cost of development and production go up the services make 
trade-offs that lead to reduced numbers of systems purchased. When the services decide 
to by fewer systems the cost per unit goes up dramatically because the contractors fixed 
costs are spread over fewer units. 

The increasing complexity of weapons systems along with the ever increasing 
demand on the federal budget has put enormous pressure on DoD acquisition programs. 
As the complexity of weapons has gone up so has the cost of the weapons. This 
complexity of weapons systems also causes the increase in development time. The 
increases to both the cost and schedule must be balanced with the performance of the 
weapon. How much performance can we afford? How long will it take to develop the 
technology to give us that level of performance? The answers to these questions provide 
the trade-offs that must be made in any program. 
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A. THE STAKEHOLDERS 

There are many people and organizations that influence the outcome of a DoD 
program. To understand the environment program managers must work within we have 
to first understand the people and organizations that influence their programs. The 
diagram below shows some of the most influential people and organizations. 



2003) 


The tortured triangle shows the three main groups that have varying degrees of 
influence over a program and how the groups are intertwined and try to exert power over 
one another. The defense industry exerts influence through the lobbying of members of 
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congress and the executive branch using tactics such as campaign contributions and 
political favors in order to influence both the executive branch and congress to win 
support for programs that produce weapon systems developed by the contractor. The 
executive branch exerts its power through control of the DoD and the services. The 
executive branch (the services and DoD) sets its priorities through the official policies 
and strategies (National Military Strategy, National Security Strategy, Quadrennial 
Defense Review, etc.) and the budget process (Planning Programming Budgeting 
Execution System, Future Years Defense Planning Guidance, Program Objectives 
Memorandum, etc.). The executive branch also exerts influence by appointments and the 
placement of key personnel to oversee the process within the DoD and the services. 
While it is called the Presidents' budget, without Congressional approval, no funding is 
authorized or allocated for the programs. This is the main control that Congress has for 
influencing weapon system acquisition. With the ever increasing complexity and cost of 
weapon systems Congress has continued to try to exert more control over the acquisition 
process. Congress has continually called for changes in the way weapons systems are 
acquired. In an attempt to control spending. Congress keeps a close watch on the DoD 
and weapon system programs. The following chart shows the weapon systems cost 
overruns faced by the DoD since the 1970s and the initiatives made by Congress, the 
Executive Branch and DoD to address problems with the acquisition process. 
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1970-1979 

1980 -1969 

1990-1999 

Development 
cost overrun: 

$13 billion 

(30%) 

Development 
cost overrun: 

$12 billion 

(39%) 

Development 
cost overrun: 

$15 billion 

(40%) 

K«y StudiM and IrritlatlvM Impacting tha Dafanaa Aoqiialtion Prooaaa 

• 1970 Fitzhugh Commission 

• 1972 Commission on Government 
Procurement 

• 1981 Carlucci Initiatives 

• 1982 Grace Commission 

1986 Packard Commission 

• 1994 Federal Acquisition Streamlining Act 

• 1996 Clinger-Cohen Act 

DOD AoqulaHion Policy Changaa 

• 1971 DOD 5000 policy established 

• 1975 Policy revised 

• 1977 Policy revised 

• 1980 Policy revised 

• 1982 Policy revised 

• 1985 Policy revised 

• 1986 Policy revised 

• 1987 Policy revised 

• 1991 Policy revised 

• 1996 Policy revised 


Soifce: [X>D (data); GAO (analysla andpr^^entadio). 


Figure 4. Development Cost Overruns by Decade (in Fiscal Year 2005 Dollars) and 
Key Reform Efforts. (From: GAO, GAO-06-368, 2006) 


The Congressional budget process and the changes to priorities often result in 
cancellations or reductions in program budgets, which result in inefficiencies in the 
procurement process. When fewer defense dollars are available, decisions must be made 
and the results are usually fewer weapons purchased over a longer period of time. This 
causes an increase to the per-unit-cost of each item and causes contractor's costs to go up 
because they are producing an inefficient quantity of weapons. 

The stakeholders listed above can have varying degrees of influence over a DoD 
weapons program, based on many factors. This research describes these organizations 
and individuals to depict the many factors that can influence a program and are outside 
the control of the PM. There are also many problems that affect a program from unstable 
funding to changes in defense priorities. These problems are usually beyond the scope of 
the program manager. The program manager must focus on what he can control in order 
to manage a successful program. This project will focus on the factors that the PM can 
control and determine which of those factors influence the success or failure of the 
program. 
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B. PROGRAM MANAGEMENT 

What is program management? Program management is the process by which the 
DoD develops current weapon systems. The DAU glossary defines program management 
as: 


The process whereby a single leader exercises centralized authority and 
responsibility for planning, organizing, staffing, controlling and leading 
the combined efforts of participating/assigned civilian and military 
personnel and organizations, for the management of a specific defense 
acquisition program or programs, through development, production, 
deployment, operations, support and disposal.6 

The concept of program management can be traced to the Air Force's ballistic missile 
development in the 1950 and 1960s and the Air Force Systems Command's development 
of publications and manuals referred to as the 375 series. In Acquiring Defense Systems, 
David Acker states that program management usually contain five distinct phases: 

(a) exploration and development of defense system concepts based upon a 
recognized mission element need; (b) demonstration and validation of 
selected alternative concepts; (c) design, development, limited production, 
test and evaluation; (d) production; and (e) Service deployment, 
operational support as well as planned and unplanned product 
improvement.^ 

This process, if executed properly ensures that every aspect of the weapon system 
from production to support to disposal has been identified and addressed to ensure the 
success of the system. While the process of program management is a complex process 
that produces a unique product, lessons can be learned from previous programs and 
implemented effectively in new programs. The majority of DoD acquisition programs 
experience unanticipated increases to both the cost and schedule of the programs. This 
research will identify internal factors from previous programs that if applied to future 
programs may help to reduce the incident of cost and schedule overruns. While the 
impact of the decisions made by the three groups of stakeholders outside the program has 


6 Defense Acquisition University, Glossary: Defense Acquisition Acronyms and Terms, 2005, 22. 

^ David D. Acker, Acquiring Defense Systems, A Quest for the Best, Defense Systems Management 
College Press, 1993. 
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a significant effect on the "success" of a program, the PM has little control over them. On 
the other hand, the internal factors can be controlled by the PM. 

C. DEFINING SUCCESS 

How do we define the success of a program? This question is an important one to 
consider when evaluating the cost, schedule and performance of a program. Can a 
program be successful if it is completed over budget? What if it doesn't have the level of 
performance that was required by the user? There have been multiple studies conducted 
by GAO, the DoD and other agencies but most fail to define the meaning of success. 
While the GAO makes the claim that program managers use a formal definition for 
success, when it comes to budgets and funding the definition becomes ambiguous. A 
GAO report made the following observation: 

At DoD, success is often formally defined in similar terms as the 
commercial world: deliver high quality products to customers (the 
warfighter) at the right time and the right cost. Virtually all program 
managers we spoke with first defined success in terms of enabling 
warfighters and doing so in a timely and cost-efficient manner. But when 
the point was pursued further, it became clear that the implied definition 
for success in DoD is attracting funds for new programs, and keeping 
funds for ongoing programs. Program managers themselves say they 
spend enormous amounts of time retaining support for their efforts and 
that their focus is largely on keeping funds stable. They also observe that 
the DoD starts more programs than it can afford to begin with, which 
merely sets the stage for competition and resulting behaviors. As noted 
earlier, there are factors that contribute to how success is defined in 
practice, including the fact that DoD depends on annual appropriations 
and it must fund a wide variety of missions beyond weapon systems 
development. 8 

Morris and Hough addressed the topic of project success in The Anatomy of Major 
Projects and they commented: "Success is a slippery concept to measure-and that it has 
different definitions depending on who you are and what your role in the project is (and 
when you attempt to measure it).^ You can be assured that the end user's definition of 
success will be different than the PM's. While the PM's criteria and user's criteria for 


° GAO, Best Practices-Better Support of Weapon System Program Managers Needed to Improve 
Outcomes, GAO-06-110, 2006, 56. 

9 Peter Morris & George Hough, The Anatomy of Major Projects: A Study of the Reality of Project 
Management, 1987, Wiley, 96. 
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success might have some overlap the end user doesn't care about acquistion cost but 
wants a functional, reliable piece of equipment that meets his needs and he wants it 
immediately. The PM has the difficult job of transforming the user's needs into the 
requirements and attempt to balance those needs with the cost and schedule. The cost, 
schedule and performance will all be impacted by the choices that are made. 

One of the major issues that this project will address is that the services the DoD, 
Congress and the GAO do not have a definition of success that is agreed upon and used 
synonymously. If the organizations can not agree on what defines success, then how can 
it be achieved? 

In Introduction of Defense Acquisition Management the author describes success 
and what it means to the four major players in the PM's environment: 

A successful system acquisition program places a capable and supportable 
system in the hands of a user when and where it is needed, and does so 
within the bounds of affordability. The ideal outcome necessary for 
successful long-term relationships among the participants in defense 
acquisition is "Win-Win", wherein each participant gains something of 
value for participating. Depending on your perspective, "success" can take 
many forms. 

• For the PM, success means a system that is delivered on time, 
within cost and meets the user's requirements. 

• For the Office of the Secretary of Defense (OSD) staff, success 
means a program that satisfies national security objectives, 
provides a balanced force structure, and does not attract undue 
Congressional scrutiny. 

• For industry, success means a program that provides a positive 
cash flow and a satisfactory return on investment, and 
preserves the contractor's competitive position in the industry. 

• For the user, success means a system that is effective in combat 
and easy to operate and maintain. 
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To a large extent, a person's (or organization's) perspeetive on what constitutes a 
successful program depends on position. In other words, where you stand on "success" is 
largely a function of where you sit.^o 

For every stakeholder in the acquisition process, "success" could have a distinct 
and different meaning. How can we expect to produce successful programs when we can 
not even agree on the definition of success? Although it is difficult to define a successful 
program in a way that would satisfy every stakeholder, it is an important question that 
must be addressed in order to create a more efficient acquisition process. 

D. CRITICAL SUCCESS FACTORS 

The theory of "Success Factors" was first developed by D. Ronald Daniel in 1961, 
but it was not until 1979 that J.F. Rockart of the Harvard Business School published work 
on a systematic approach to Critical Success Factors (CSF) in "A Primer on Critical 
Success Factors". His work focused on factors within organizations that if identified 
and measured can produce successful results. Rockart defined CSF: 

...the limited number of areas in which results, if they are satisfactory, 
will ensure successful competitive performance for the organization. They 
are the few key areas where things must go right for the business to 
flourish. If results in these areas are not adequate, the organization's efforts 
for the period will be less than desired, i^ 

While Rockart's research focused on identification of CSF at the executive level. 
Dr. James Dobbins and Dr. Richard Donnelly published research in 1998 regarding CSF 
within government acquisition programs. 13 Their work published in Acquisition Review 
Quarterly focused on identification of CSFs within government acquisition programs that 
can help PMs manage successfully. Dr. Dobbins published several more articles 
regarding CSFs and their application within the DoD acquisition process and risk 
management. 


Defense Acquisition University, Introduction to Defense Acquisition Management, 1999, 7. 

James H. Dobbins & Richard G. Donnelly, Summary Research Report on Critical Success Factors 
in Federal Government Program Management, Acquisition Review Quarterly Winter-98, 1998, 62. 

12 James H. Dobbins & Richard G. Donnelly, Summary Research Report on Critical Success Factors 
in Federal Government Program Management, Acquisition Review Quarterly Winter-98, 1998, 62. 

13 Ibid. 
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E. SUMMARY 

This chapter has briefly deseribed the organizations and people assoeiated with 
the DoD aequisition proeess. It has also demonstrated the external issues that the PM 
must deal with while managing a weapon system program. Since World War II weapon 
systems have become more eomplex and thus more expensive. In order to manage those 
programs the serviees assigns a PM to manage the weapon system programs. The PM 
must deal with three main organizations or groups (Executive, Congress, Industry) 
throughout the life of the program. There are many faetors that affeet a program both 
externally and internally. The subsequent ehapters will identify the internal faetors that 
ean have a negative affeet on weapon system programs. There are many ehallenges that 
faee the Program Manager of a major weapon system program. He is faeed with a system 
that is extremely eomplex to produce. He is also faced with a multitude of people and 
organizations that have input into his system and the PM must try to exert some control 
over these people and organizations. Unlike the eommereial world, PMs within the DoD 
exert far less eontrol over their programs. The next ehapter will focus on the researeh that 
has been eondueted regarding Critieal Sueeess Faetors by Dobbins and others and its 
applieation to DoD weapon aequisition programs. 
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III. RESEARCH ON SUCCESS EACTORS 


A. DOBBINS' AND DONNELLY'S RESEARCH 

In 1998 James Dobbins and Richard Donnelly published research regarding 
Critical Success Factors and the implications on government acquisition programs. Prior 
to their work, most research on CSFs was conducted on for profit businesses in the 
private sector. This research focused on the strategic level, namely executive level 
managers within specific industries. The basic premise of Dobbins' and Donnelly's work 
was to determine if it was possible to identify general CSFs that were relevant to most 
major government acquisition programs. Dobbins and Donnelly sought to answer the 
following questions, "Are there any general Critical Success Factors for DoD 
programs?" They surmised that the identification of CSFs that are common to DoD 
acquisition programs and the identification of metrics to correctly measure the CSFs will 
help future PMs manage their programs with a higher degree of success. 

Dobbins and Donnelly used a survey that was developed based on the CSF 
categories originally identified by Bullen.i^ The survey asked DoD PMs to identify CSFs 
in the related categories and the associated metric for the identified CSF. The survey was 
mailed to two separate groups of PMs. The first group consisted of PMs managing 
embedded system programs and the second group was PMs managing automated 
information systems. Dobbins and Donnelly distributed the surveys to both groups 
including 73 to the embedded system PMs and 57 to the automated information system 
PMs. They received back 20 completed surveys from the embedded system PMs and 14 
from the automated information system PMs. The relatively low percentage of returned 
surveys, 27% and 25% respectively, is typical for surveys sent to PMs due to the many 
official and unofficial questionnaires in which PMs are asked to respond. 

The results showed 18 CSFs that were common to both groups of PMs. The 
following three charts show the results for the two groups and the 18 combined CSFs that 


James H. Dobbins & Richard G. Donnelly, Summary Research Report on Critical Success Factors 
in Federal Government Program Management, Acquisition Review Quarterly Winter-98, 1998, 61. 

15 Ibid., 63. 
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were identified. The top four CSFs are highlighted and will be analyzed further in 
Chapter 4. It is interesting to note is that the program managers specifically mentioned 
establishing systems engineering expertise within the program office as it related to 
having a technically competent program staff. 


CSF 

Times 

N = 20 Factor 

12 

15 

Continuous meaningful visibility using measures 

4 

9 

Technically competent program office staff 

2 

9 

Clearly defined and stable requirements 

1 

8 

Stable and adequate funding 

3 

5 

Risk management 

7 

5 

Schedule management 

15 

5 

Stable, qualified industrial base 

17 

5 

Effective vertical a lateral communication 

16 

4 

Management political influencing agents 

6 

3 

Stable and adequate personnel resources 

8 

3 

Cost management 

9 

3 

User involvement, support and acceptance 

10 

3 

Strong and structured quality control 

11 

2 

Clearly and objectively defined project goals 

19 

2 

Development and execution of program management strategic plan 

22 

2 

Change management 

5 

1 

Configuration management and control 

13 

1 

Other agency support for training and government furnished 
equipment (GEE) 

14 

1 

Adequate program office resources 

18 

1 

Eeadership 

20 

1 

Thorough system documentation 

21 

1 

Test and evaluation master plan (TEMP) approval 

23 

1 

Program office teamwork 

24 

1 

Effective and timely decision making 

25 

1 

Eoreign military sales 

26 

1 

Measure and control integrated logistics support performance 

27 

1 

Initiation of new projects 


Table 1. CSFs Identified by Embedded Systems Program Managers in Priority Order 
(From: Summary Research Report on Critical Success Factors in Federal 
Government Program Management, 1998) 


James H. Dobbins & Richard G. Donnelly, Summary Research Report on Critical Success Factors 
in Federal Government Program Management, Acquisition Review Quarterly Winter-98, 1998, 68. 
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CSF 1 

Times 

1 N = 14 Factor 

1 

10 

Stable and adequate budget 

9 

9 

User involvement and support 

12 

9 

Effective technical performance evaluation 

2 

8 

Detailed requirements analysis 

4 

8 

Technically competent staff 

19 

7 

Top management support 

17 

6 

Effective vertical a lateral communication 

7 

6 

Schedule management 

10 

6 

Strong quality control program 

6 

5 

Stable project staff 

16 

5 

Management political influencing agents 

13 

4 

Other agency support for training and government furnished equipment (GEE) 

3 

3 

Risk management 

20 

3 

Strong knowledge of life cycle management 

23 

3 

Incremental acquisition 

8 

2 

Cost management 

22 

2 

Common Sense 

11 

2 

Clearly defined mission 

14 

2 

Adequate program office resources 

21 

1 

Objective economic analysis 

18 

1 

Leadership 

15 

1 

Stable, qualified industrial base 

5 

1 

Configuration management and control 

24 

1 

On-site team to prevent fraud, waste and abuse 


Table 2. CSFs Identified by Automated Info Systems Program Managers in Priority 


Order (From: Summary Research Report on Critical Success Factors in 
Federal Government Program Management, 1998) 


CSF 

Times 

1 N = 34 Factor 

12 

24 

Continuous meaningful visibility using measures 

1 

18 

Stable and adequate funding 

18 

2 

Leadership 

2 

17 

Clearly defined and stable requirements 

4 

17 

Technically competent program office staff 

9 

12 

User involvement, support and acceptance 

7 

11 

Schedule management 

17 

11 

Effective vertical a lateral communication 

10 

9 

Strong and structured quality control 

16 

9 

Management political influencing agents 

3 

8 

Risk management 

6 

8 

Stable and adequate personnel resources 

15 

6 

Stable, qualified industrial base 

13 

5 

Other agency support for training and government furnished equipment (GEE) 

8 

5 

Cost management 

11 

4 

Clearly and objectively defined project goals 

14 

3 

Adequate program office resources 

5 

2 

Configuration management and control 


Table 3. Combined CSFs Identified by Both Groups of Program Managers (From: 

Summary Research Report on Critical Success Factors in Federal Government 

Program Management, 1998) 


17 













Dobbins and Donnelly concluded that CSFs in DoD acquisition programs were 
identifiable and published the following findings: 

• The CSFs for DOD program management are identifiable, and 
their explicit identification would clearly assist the program 
managers in maintaining management focus on the factors most 
important to program success. 

• A significant number of CSFs are common to both groups of 
program managers. 

• The component assumptions and emphasis for a given CSF 
common to both groups may be slightly different. This difference 
is largely a function of the difference between the mission of the 
two groups, the embedded system PMs being more concerned with 
the complete development of total systems than the automated 
information system PM. 

• The CSFs identified by the PMs as the most significant for 
program success are not those factors that received the most 
attention from the oversight activities and agencies. 

• The measures identified most often by the program managers as 
those used or recommended for use, are significantly more oriented 
toward cost and schedule (which must be briefed to oversight 
agencies) rather than toward factors identified by the program 
managers in the field as being most critical to the program success. 

• There is no widely recognized and generally used set of measures 
consistent with the most frequently reported CSFs. This suggests 
that even though various factors are recognized as critical, they are 
not usually explicitly identified and the information network 
required to manage against those critical factors is not well 
developed. 

• A commonly recognized set of CSFs, and a consistent 
measurement-based information network based on these CSFs, 
would be of significant benefit to the program managers as well as 
the oversight agencies. Such a management system would 
significantly improve the management success potential on 
programs across the board, and would provide the external groups 
a consistent method for evaluating and comparing different 
programs so that recommendations for future improvements could 
be results-based. 

• A CSF-based information network for program management 
would lend itself to not only increased visibility for the program 
manager and staff during all life cycle phases, but would provide 
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the base for the establishment of measures for determining when 
the underlying assumptions for a given CSF may be changing. 

• A CSF-based information network would provide a common 
framework for productive discussions between the program 
manager and the external groups, including the development 
contractors and would greatly support the effectiveness of IPTs. 

• A CSF-based information network would significantly reduce the 
duplicative reporting and diversions the program managers 
experience under the present conditions. 

• A CSF-based analysis process would be significant teaching 
instrument for educating prospective program managers in 
strategic thinking in terms of those considerations critical to 
success. 1”^ 

Other research and studies relating to the DoD have added to the body of knowledge 
regarding CSFs and their importance to DoD weapon system acquisitions. 

B. GAO-06-110 BEST PRACTICES 

The GAO published data regarding better support of weapons systems in the best 
practices series titled "Better Support of Weapon System Program Managers Needed to 
Improve Outcomes" (GAO-06-110). This GAO report used the case study methodology 
to identify the best practices and processes used by commercial companies to ensure their 
programs were successful. The GAO selected companies that produced complex products 
using program management to conduct research, development and production. The GAO 
studied three companies; Toyota, Siemens and Motorola, conducting interviews with 
senior leaders and program managers to identify the processes, practices and metrics that 
were used by senior management to support program managers and hold them 
accountable for their program. In addition, the GAO conducted a survey of current PMs 
of ACAT I and II programs. The survey was sent to 185 PMs of which 128 responded 
resulting in a 69 percent response rate. The GAO also conducted in-depth interviews with 
individual PMs and Program Executive Officers as well as PMs from Boeing and 


James H. Dobbins & Richard G. Donnelly, Summary Research Report on Critical Success Factors 
in Federal Government Program Management, Acquisition Review Quarterly Winter-98, 1998, 77. 
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Lockheed Martin managing two major weapon systems. They also synthesized 
information from previous GAO work regarding best practices in product development, 

The GAO found many differences between the programs being run by successful 
companies and DoD acquisition programs. The following topics were discussed as major 
differences between commercial companies studied by the GAO and the DoD acquisition 
process. 

1. Funding 

One of the key issues noted in the GAO report was the competition for funding. 
While the DoD is very good at developing long-term strategic plans, those plans do not 
appear to lead to a long-term realistic investment strategy including weapon 
development. Instead, there is severe competition because more programs are started than 
can be funded, this in turn, leads to increased competition and overly optimistic estimates 
regarding cost, schedule and performance of the competing programs. The GAO has 
continually identified problems with the DoD budget process including this assessment in 
2003: 


Since the mid-1980s, we have reported that the DoD employs overly 
optimistic planning assumptions in its budget formulation. As a result, 

DoD has too many programs for the available dollars, which often leads to 
program instability, costly program stretchouts, and program termination. 

In 2000, we reported that because the fiscal year 2001 program's projected 
cost was $16 billion more than the cost projected for the same elements in 
fiscal year 2000 program, DoD could not implement its operation and 
maintenance and procurement programs as planned. Over the past few 
years, the mismatch between programs and budgets has continued, 
especially in the area of weapon systems acquisition. 

The problems identified by the GAO have also been identified by DoD program 
managers. These comments were taken directly from program managers interviewed by 
the GAO: 


Government Accountability Office, Best Practices-Better Support of Weapon System Program 
Managers Needed to Improve Outcomes, GAO-06-110, 2006, 20. 

GAO, Major Management Challenges and Program Risks, GAO-03-98, 2003, 11. 
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• OSD staff has reduced funding without any understanding or 
appreciation for program impacts. It appears that the staff 
makes arbitrary cuts. 

• OSD has a very near-term execution year focus, resulting in 
great instability. In reality, it should provide much more 
strategic vectors for the Department instead of short-term 
adjustments to fix more tactical-level funding needs. 

• My experience is that the service and OSD typically cut 
programs to pay top down bills. 

• There is no such thing as funding stability in DoD. Funding 
reductions and programs stretchouts are the norm due to top 
down fiscal bills that occur during the execution year. The 
pentagon must pay the bills, therefore it takes funds from the 
programs, thereby contributing to program stretchout, cost 
increases, inefficiencies, etc. 

• Unstable funding results in pressure to do aggressive things in 
order to minimize the impact of budget cuts on schedule and 
performance. I believe this has been a major factor in 
recent.. .program execution problems. 

• Our product is considered a support function. When funding 
gets tight, we have been considered a bill payer for others, even 
if it has "broken" our program. 20 

The next chart depicts the results when the GAO asked DoD program managers 
about the obstacles they faced. 


20 Government Accountability Office, Best Practices-Better Support of Weapon System Program 
Managers Needed to Improve Outcomes, GAO-06-110, 2006, 40. 
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Figure 5. How Program Managers Responded to an Open-ended Question on What 
Were the Biggest Obstacles They Faced (From: GAO, GAO-06-110, 2006) 

In contrast, key leaders within the companies the GAO researched, ensured that 
the projects undertaken supported the investment strategy of the company. This process 
gives the assigned program manager confidence that the leadership is committed to the 
program and there is a clear understanding of funding priorities. It is interesting to note 
that the GAO found in their research that not one program manager, from any of the 
companies, ever mentioned funding as a problem during the program, funding was a 
given once senior leaders had committed to their project.2i 

2. Requirements 

The companies the GAO visited achieved there overall investment strategy by 
matching requirements to the available resources. All the key resources needed, including 

Government Accountability Office, Best Practices-Better Support of Weapon System Program 
Managers Needed to Improve Outcomes, GAO-06-110, 2006, 25. 
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time, money, technology and people, were available before the program began. The 
companies had already conducted extensive research and all requirements were clearly 
defined prior to initiation and commitment to the program. Previous research conducted 
by the GAO in the best practices series had consistently found that matching 
requirements and resources prior to initiating a new program to be a hallmark for 
successful companies. While research on DoD programs noted that requirements were 
often not fully defined at the onset of a program, and many also pointed out that users and 
stakeholders often did not adhere to the agreements made when programs were launched, 
especially if technologies did not mature as planned. 22 

3. Staffing 

The GAO report 06-110 identified staffing issues as an important concern of both 
program managers and program executive officers within the DoD. They pointed to 
critical shortages of staff in the areas of program management, systems engineering, cost 
estimating and software development.23 The GAO also identified practices within 
successful companies that produced highly qualified program managers. The companies 
placed a high value on strong leadership qualities including, decision making skills, 
diplomacy, communication skills and the ability to motivate others. The program 
managers interviewed by the GAO had a combination of formal training and informal 
mentorship by senior executives. The companies that were visited also had formal 
process for developing and deploying experts to assist the program manager. The 
program managers had a high degree of confidence in their support staff. 

C. DELANO'S RESEARCH 

Further research was published in the winter 1998 issue of Acquisition Review 
Quarterly (now Acquisition Review Journal) that supported the idea of Critical Success 
Factors. Major Kenneth Delano's research concluded that there were factors in DoD 
acquisition programs that could be identified as being key to the success of programs. 
Delano's research was conducted using survey questions and a literature review. Of the 
32 surveys sent to PMs, 18 were returned. This resulted in a 56 percent rate of return 

22 Government Accountability Office, Best Practices-Better Support of Weapon System Program 
Managers Needed to Improve Outcomes, GAO-06-110, 2006, 42. 

23 Ibid, 44-45. 
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which was sufficient to validate the survey results. The next table depicts the results of 
the survey. The highlighted areas within the tables will be analyzed further in Chapter 4. 


Average Score 

Program Success Factor 

4.42 

Program Manager's ability to communicate 

4.25 

Type and quality of people associated with the program 

4.25 

Program managers ability to lead 

4.25 

Good relationship with user organization 

4.17 

Resources: people, facilities, money 

4.08 

Product requirements and design stability 

3.91 

Funding stability 

3.83 

Good relationship with prime contractor 

3.58 

Program's acquisition strategy 

3.58 

Program manager's acquisition experience 

3.25 

Program personnel continuity 

3.00 

Program manager continuity 

3.00 

High degree of technical difficulty 

2.92 

Program managers field experience 

2.67 

Program managers technical ability 

2.33 

Total quality management program 

2.25 

Low degree of technical difficulty 

5 = Critical Factor 

3 = Important Factor I = Not Very Important Factor 

4 = Very Important Factor 2 = Somewhat Important Factor 


Table 4. Delano's Program Success Factors Rank by Importance (From: Identifying 

Factors That Contribute to Program Success, 1998) 


Delano's literature review consisted of researching 19 articles, documents and 
books. He then categorized key factors into two groups; Acquisition Factors and 
Resource Factors. These categories were then measured as to the number of times the 
factor was mentioned in an article and he concluded through his calculations, that factors 
with a correlation higher than 32 percent were considered the most significant.^^ While 
the factors that Delano used do not exactly mirror the factors Dobbins used there are 
many correlations and enough data to compare the two. The results of Delano's literature 
review are shown in the following chart. 


Kenneth J. Delano, Identifying Factors That Contribute to Program Success, Acquisition Review 
Quarterly Winter-1998, 1998, 42. 
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Factor 


Source 


Total 


% 



A 

B 

C 

D 

E 

F 

G 

H 

I 

J 

K 

L 

M 

N 

O 

p 

Q 

R 

S 



Well defined 
requirements 

X 


X 

X 

X 


X 

X 



X 


X 

X 






9 

47 

Acquisition 

strategy 

X 

X 


X 

X 



X 

X 


X 





X 


X 


9 

47 

Works well 
when fielded 

X 







X 

X 


X 





X 


X 


6 

32 

Stability 

X 


X 

X 

X 



X 








X 




6 

32 

Good 

relations 

with 

contractor 

X 




X 


X 








X 





4 

21 

TQM 

program 








X 









X 


X 

3 

16 

Meets 

performance 

objectives 

X 



X 
















2 

11 

Meets cost 
objective 




X 
















1 

5 

Meets IOC 
date 




















0 

0 


Table 5. Delano's Acquisition Factor Analysis (From: Identifying Factors That 

Contribute to Program Success, 1998) 


Factor 

Source 

Total 

% 


A 

B 

C 

D 

E 

F 

G 

H 

I 

J 

K 

L 

M 

N 

o 

p 

Q 

R 

S 



Quality 

people 

X 


X 

X 

X 



X 





X 

X 

X 



X 


9 

47 

PM 

responsibility 
& authority 

X 


X 

X 

X 


X 





X 


X 


X 




8 

42 

Total team 
concept 

X 




X 


X 




X 

X 


X 

X 





7 

37 

PM skills 

X 



X 

X 



X 




X 

X 

X 






7 

37 

Congressional 

involvement 




X 

X 









X 






3 

16 

User 

involvement 







X 





X 








2 

10 

Adequate 

resources 



X 





X 












2 

10 

Adequate 

staffing 















X 





1 

5 
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Factor 

Source 

Total 

% 

Spt agency 
involvement 







X 













1 

5 

Higher Cmd 
involvement 







X 













1 

5 

PM technical 
ability 

X 



















1 

5 

GAO 

involvement 




















0 

0 


Table 6. De 


ano's Resource Factor Analysis (From: Identifying Factors That 
Contribute to Program Success, 1998) 


A 

Baumgartner, Brown and Kelly 

B 

Beltramo 

C 

Clay 

D 

Gansler 

E 

Gregory 

F 

Heberling and Graham 

G 

Hicks, Rich, Wertheim and Meyer 

H 

Hirsch and Waelchli 

I 

Kish 

J 

Lesser 

K 

Nelson 

L 

Price and Valentine 

M 

"RX for Ailing Procurement System" 

N 

Sammet and Green 

0 

Settlmeyer 

P 

Snoderly and Acker 

0 

Total Quality Management Master Plan 

R 

Weiss 

S 

Zairi 


Table 7. Correspondence Between Codes and Sources (From: Identifying Factors That 


Contribute to Program Success, 1998) 

Delano's research concluded that the following factors were ranked high both in the 
literature review and the PM survey and considered key to the success of DoD acquisition 
programs. 


• Well defined requirements 

• Acquisition strategy 

• Actual weapon system performance 
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• Program stability 

• Quality people 

• PM authority 

• Total team concept 

• PM skills^s 

While the data that is presented in Delano's paper has distinct differences there are 
many similarities to the research done by Dobbins and Donnelly. Some of the differences 
in the survey results could be explained by the questionnaire format and focus of the 
surveys, but the main point of Delano's research is the same as Dobbins' and Donnelly's, 
that there are factors common to most programs that if they are managed and controlled 
will lead to a successful program. 

D. RAND STUDY 

In 1996, Robert Johnson and John Birkler published "Three Programs and Ten 
Criteria" which was sponsored by the Office of the Secretary of Defense (OSD) and 
addressed issues within DoD acquisition management. The authors used their 
observations combined with lessons learned from past programs, as well as their 
knowledge of DoD acquisition management to develop a structured list of "key factors" 
to be considered in successfully managing a major defense acquisition program. The 
following is a list of factors that Johnson and Birkler identified as critical to the success 
of a weapons system program: 

• Lines of authority have been established and are clear. Defense 
Management Review issues and/or problems must not cause 
confusion, bickering, or a diminution of Program Manager 
responsibility and accountability. 

• Communication is open (no secrets-all information is divulged; 
using all media and avenues, e.g., email, written, verbal) and 
continuous at and between all levels of authority. 


Kenneth J. Delano, Identifying Factors That Contribute to Program Success, Acquisition Review 
Quarterly Winter-1998, 1998, 43. 
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• Cost/Schedule Control System (CS2), cost performance 
measurement (CPM), and other management reports are used as 
indicators of trends in program progress and for reporting program 
status. 

• Risk-management techniques have been implemented. 

• Program stability has been achieved through control of 
requirements. 

• A strong Government-industry support team (Program Office, 
functional support, Defense Plant Representative Offices) is 
present and has explicit mechanisms for coordinating 
responsibilities. 

• Incentives for the Program Manager are adequate and positive. 

• Funding is stable and adequate. 

• Selection of best-qualified personnel for key acquisition 
management positions is objective and regulated. 

• Security requirements do not restrict adequate and sufficient 

management.26 

The author used this list of factors to form a baseline for a well managed program and see 

the above factors as critical to managing a successful acquisition program. The author's 

main points listed above focus on the same areas as research conducted by Dobbins and 

Donnelly as well as Delano and find that areas such as requirements, funding, risk 

management and staffing are critical to the success of DoD acquisition programs. 

E. LESSONS LEARNED FROM THE F/A-22 AND F/A-18E/F 
DEVELOPMENT PROGRAMS 

At the behest of the Air Force the Rand Corporation conducted a review of the 
F/A-22 and F/A-18E/F programs in order to identify lessons learned that could be applied 
to future programs. The results were published in 2005 and provide useful information to 


Robert V. Johnson and John Birkler, Three Programs and Ten Criteria-Evaluating and Improving 
Acquisition Program Management and Oversight Processes Within the DoD, 1996 Rand, 7-8. 
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future program managers regarding critical factors and their impact on the outcome of 
DoD acquisition program. The Rand study identified the following lessons learned: 


• Early, realistic cost and schedule estimates set the program on the 
right path for the rest of the development program. 

• A stable development team structure, proper team expertise, clear 
lines of responsibility and authority and a lead contractor 
responsible for overall program progress are critical to program 
success. 

• An experienced management team and contractors with prior 
business relationships help eliminate early management problems. 

• Concurrent development of new technology for the airframe, 
avionics and propulsion adds significant risk. 

• Reducing the cost and risk of avionics should be a key focus of the 
concept development phase. Avionics is a considerable cost driver 
of modem weapon systems, and new concepts should be 
demonstrated along with new airframe designs. 

• Preplanned, evolutionary modernization of high-risk avionics can 
reduce risk and help control cost and schedule. 

• Careful monitoring of airframe weight is important. Airframe 
weight instability is an early indicator of problems. 

• Earned Value Management (EVM) data should be used to monitor 
and manage program costs at the level of integrated product teams 
(IPTs). 27 

F. GAO REPORTS 

Several GAO reports are also relevant to the research on Critical Success Eactors 
within DoD Acquisition programs. While the data provided in these reports does not 


27 Obaid Younossi, David E. Stern, Mark Lorell, Frances M. Fussier, Lessons Learned from F/A-22 
and F/A-18 E/F Development Programs, 2005, Rand, 57. 
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specifically address Critical Success Factors these GAO reports present separate issues 
that programs have encountered during their development that are closely tied to Critical 
Success Factor Theory. 

1. GAO-06-585T Defense Acquisitions-Actions Needed to Get Better 

Results on Weapon Systems Investments 

This document outlines the major issues with DoD acquisitions from the GAO's 
perspective and then provides solutions that it believes will address the problems that the 
DoD faces. Several of the problems they identify and solutions they suggest are similar to 
those that are addressed by the Critical Success Factors. The first issue, is the problem of 
well defined and stable requirements. Regarding requirements, the GAO stated: 

...DoD has exacerbated their problems by not clearly defining and 
stabilizing requirements before programs are started. At times, in fact, it 
has allowed new requirements to be added well into the acquisition cycle- 
significantly stretching technology and creating design challenges, and 
exacerbating budget overruns. For example, in the F-22A program, the Air 
Force added a requirement for air-to-ground attack capability. In its 
Global Hawk program, the Air Force added both signal intelligence and 
imagery intelligence requirements. 

In order to address the problem the GAO recommends that system requirements are 
agreed by the service acquisition executives and the warfighters and that no additional 
requirements are added during execution unless they are fully funded. 

Another issue that the GAO addresses is funding of weapons systems. The GAO 
report states: 

DoD starts more weapons programs than it can afford and sustain, creating 
a competition for funding that encourages low cost estimating, optimistic 
scheduling, over promising and suppressing of bad news. Programs focus 
on advocacy at the expense of realism and sound judgment. Invariably, 
with too many programs in its portfolio, DoD and the Congress are forced 
to continually shift funds to and from programs.^9 


Government Accountability Office, Defense Acquisition-Actions Needed to get Better Results on 
Weapons Systems Investments, GAO-06-585T, 2006, 6. 

^9 Government Accountability Office, Defense Acquisition-Actions Needed to get Better Results on 
Weapons Systems Investments, GAO-06-585T, 2006, 6. 
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Their solution is to enforce funding for priorities annually and measure success 
against the original plan. The GAO report addresses many other issues outside 
the scope of this research which is not pertinent to the discussion on Critical 
Success Factors. 

2. GAO-03-98 Major Management Challenges and Program Risks 

This report was written as part of a series on performance and accountability and 
identifies systemic and specific problems with management processes. While this report 
does not address Critical Success Factors directly, some of the systemic issues the GAO 
found within the DoD have implications with regard to CSFs. Specifically, The GAO 
findings regarding technical risk and critical technology maturity. The GAO report stated, 
"...we have found major weapon systems at risk of not being able to meet program 
objectives because critical technologies were immature and software development was 
not effectively managed, The systems that are at risk because of immature technology 
include the Joint Strike Fighter, the Airborne Laser and the Space-based Infrared System. 
The report focuses on several management challenges but the main point regarding 
Critical Success Factors is that risk management and specifically technological risk is a 
major concern for weapon system programs. 

3. GAO-04-393 Defense Acquisitions-Stronger Management Practice are 
Needed to Improve DoD's Software-Intensive Weapon Acquisitions 

This GAO report focused on problems related to DoD's management of software 

and software development within weapon systems. The GAO visited three leading 

commercial software developers to identify procedures that could improve DoD's 

software development processes. While many issues are addressed in this report, the 

section regarding requirements is relevant to the discussion of Critical Success Factors. 

The GAO found: 

Senior managers at software development and acquisition companies we 
visited expect requirements to be managed and controlled before design 
work begins and virtually all lower-level design elements to be adequately 
defined before the start of coding. Without adequate definition and 
validation of requirements and design, software engineers could be coding 

30 Government Accountability Office, Major Management Challenges and Program Risks, GAO-03- 
98, 2003, 54. 
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to an incorrect design, resulting in missing functionality or errors. 
Motorola CSG, a communications company, and Teradata, a division of 
NCR that specializes in database technology, estimate that 95 percent of 
their requirements are set by the end of the requirements phase and 98 
percent by the end of the design phase. Officials view managing 
requirements as the most critical development task to ensure successful 
software outcomes. 

The report found that stable and defined requirements are critical to the success of any 

software development process. As weapon systems become more complex and dependent 

on software the ability to stabilize and clearly define requirements becomes even more 

crucial for the successful acquisition of weapons. 

4. GAO-03-55 Acquisition Workforce - Status of Agency Efforts to 

Address Future Needs 

This report focuses on problems regarding the size and quality of the workforce 
within Government organizations. The GAO identifies the work the DoD has done on 
human capital strategy as a model for other organizations to follow. The GAO 
acknowledges that the DoD has room for improvement but is farther along than most 
organizations and stated the following, "DOD has been working for several years to 
strengthen its civilian acquisition workforce. The acquisition workforce comprises a large 
proportion of the overall workforce, and DOD views the acquisition workforce as critical to 
accomplishing its mission. "32 

Improving the quality of the DoD workforce has been part of the strategy within the DoD 
for many years and incorporates guidance from the Defense Acquisition Workforce Improvement 
Act (DAWIA), the Clinger-Cohen Act and other policies to ensure the acquisition workforce has 
the necessary skills that are required to assist in the successful acquisition of weapon systems. 
While the DoD has identified the need for a professional workforce the downsizing of the same 
workforce has caused several problems, including the following reported by tbe GAO: 

During the past decade, DOD has downsized its civilian acquisition 
workforce by half. It now faces what it considers to be serious imbalances 
in the skills and experience of its remaining workforce and the potential 
loss of highly specialized knowledge if many of its acquisition specialists 

31 Government Accountability Office, Defense Acquisitions-Stronger Management Practices are 
Needed to Improve DoD's Software-Intensive Weapon Acquisitions, GAO-04-393, 2004, 14. 

32 Government Accountability Office, Acquisition Workforce-Status of Agency Effort to Address 
Future Need, GAO-03-55, 2003, 16. 
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retire. DOD created the Acquisition 2005 Task Force to study this problem 
and develop a strategy to replenish personnel losses. The task force’s first 
recommendation was to develop and implement a human capital strategic 
plan for the civilian acquisition workforce. 33 

The report identified the need to have a well-trained, professional and certified workforce 
that has the right mix of skills and is sustainable in order to assist their organizations to 
be effective. 

G. CONCLUSION 

This chapter presented data related to Critical Success Factors from a variety of 
sources including the Government Accountability Office, Rand Corporation and several 
authors. The next chapter will illustrate that Critical Success Factors are relevant to DoD 
acquisition programs and can be used to successfully manage programs. Furthermore, it 
will show the relationship between the variety of research that has been done by the 
different authors and organizations. 


33Government Accountability Office, Acquisition Workforce-Status of Agency Effort to Address 
Future Need, GAO-03-55, 2003, 17. 
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IV. ANALYSIS OF CRITICAL SUCCESS FACTORS 


There is significant data regarding the failure of DoD acquisition programs and 
what steps the DoD and services should take to "fix" the problems with the acquisition 
system. For example, the GAO has complete series of reports dedicated to improving 
DoD acquisition, such as the Best Practices Series which identifies industry leaders and 
applies there best practices to government applications. The data in chapter three is an 
attempt by professionals and government organizations to identify issues that lead to the 
failure of a program and identify corrective actions in order to create a more favorable 
outcome with future DoD programs. The data gathered comes from many sources 
including, lessons learned from past programs, this includes reports such as Three 
Programs and Ten Criteria^^, interviews with program managers, such as those conducted 
by the GAO for the report GAO-Ob-llO^^, interviews and studies of program managers 
and programs in the civilian sector as well as professional experience, such as the Rand 
report Lessons Learned from the F/A-22 and F/A-18 E/F Development Programs.36 The 
following factors have been identified by several authors or publications as critical to the 
success of programs and support Dobbins conclusions that there are Critical Success 
Factors common to most, if not all DoD acquisition programs. Dobbins' and Donnelly's 
research found 18 CSFs that they considered critical to successful programs and relevant 
to all DoD programs. While most other data does not find the exact results that Dobbins 
and Donnelly did, other research has found elements with key similarities. For example, 
Dobbins and Donnelly use the category of "Risk Management" for one of their Critical 
Success Factors, while Delano divides risk management into several categories including 
"High degree of technical difficulty" and "Program's acquisition strategy". These 
categories are not identical but are similar when they are applied to acquisition programs. 


34 Robert V. Johnson and John Birkler, Three Programs and Ten Criteria-Evaluating and Improving 
Acquisition Program Management and Oversight Processes Within the DoD, 1996 Rand. 

33 Government Accountability Office, Best Practices-Better Support of Weapon System Program 
Managers Needed to Improve Outcomes, GAO-06-110, 2006. 

36 Obaid Younossi, David E. Stern, Mark Lorell, Frances M. Lussier, Lessons Learned from F/A-22 
and F/A-18 E/F Development Programs, 2005, Rand. 
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A. STABLE AND ADEQUATE FUNDING 

In Dobbins and Donnelly's research among DoD Program Managers stable and 
adequate funding is the number one Critical Success Factor identified by both the 
Embedded System PMs and the Automated Information Systems PMs.^'^ The GAO found 
the same results when interviewing PMs, and Figure 4 clearly illustrates that PMs 
identify funding instability as the number one issue when addressing DoD acquisition 
management.38 Delano's research as well as Johnson and Birkler's also found that stable 
funding was critical to the success of acquisition programs. With regard to funding 
stability the Johnson and Birkler stated, "Budget instability plagues all three programs 
(F/A-18EF, F-22, RAH-66) and causes the greatest concern for acquisition-management 
officials. Failure to meet this criterion was the most seriously detrimental aspect we 
found during our research on the three programs."39 The GAO found that among PMs in 
the private sector stable funding was not a serious concern during the life of a program. 
The private sector PM must worry about the cost of his program and ensure that the cost 
is within a narrow band of the cost estimates but once a strategic decision is made for a 
new product, the PM has the support and understands that the senior leaders are fully 
committed to the program. 

B. CLEARLY DEFINED AND STABLE REQUIREMENTS 

As presented in the previous chapter, Dobbins' and Donnelly's research found that 
stable requirements was ranked second among CSFs and was mentioned 17 times in the 
returned surveys.^o Their research found that clear and stable requirements were critical 
to the success of acquisition programs. 

Further research by the GAO and others has also shown that requirements are 
critical to the success of programs. The GAO report 06-110 found that DoD program 


3^ James H. Dobbins & Richard G. Donnelly, Summary Research Report on Critical Success Factors 
in Federal Government Program Management, Acquisition Review Quarterly Winter-98, 1998, 69-70. 

3^ Government Accountability Office, Best Practices-Better Support of Weapon System Program 
Managers Needed to Improve Outcomes, GAO-06-110, 2006, 44. 

39 Robert V. Johnson and John Birkler, Three Programs and Ten Criteria-Evaluating and Improving 
Acquisition Program Management and Oversight Processes Within the DoD, 1996 Rand, 55. 

James H. Dobbins & Richard G. Donnelly, Summary Research Report on Critical Success Factors 
in Federal Government Program Management, Acquisition Review Quarterly Winter-98, 1998, 71. 
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managers saw requirements instability as a major obstacle to their program. Program 
managers interviewed by the GAO made the following comments: "...that requirements 
were often not fully defined at the onset of the program and stakeholders often did not 
stick to the agreements they made when programs were launched, especially if 
technologies did not mature as planned."^^ When DoD program managers were asked to 
respond to the open-ended question regarding the biggest obstacle to their program, 
requirements instability was mentioned often and only less than funding instability as a 
major concern.42 The Report also showed that, in companies that the GAO reviewed, 
requirements were clearly identified and the senior leadership ensured that the company 
had the resources to meet the requirements set out in the program. The companies will 
only fund programs with clearly defined and stable requirements. While the GAO 
reported the following regarding DoD programs, "...program managers commented that 
requirements continue to be added as the program progresses and funding instability 
continues throughout. These two factors alone cause the greatest disruption to programs, 
according to program managers. "43 

Major Delano's research also concluded that product requirements and design 
stability was considered very important among the respondents to his survey. As the 
tables in the previous chapter showed. Product Requirements and Design Stability (4.08 
out of 5.00) was considered very important in the success of a program. Delano's 
literature review also had requirements ranked high among the books, articles and 
journals he examined. In his analysis of acquisition factors Delano found that "well 
defined requirements" was mentioned 47 percent of the time. He found this to be 
significant and determined that it is critical to the success of a program. From the data 
presented in the previous chapter, there appears to be a correlation between successful 
weapon systems programs and clearly defined and stable requirements. 

While it seems simple to say that DoD programs need clearly defined and stable 
requirements, in practice, it is difficult to achieve. DoD programs often experience 

41 Government Accountability Office, Best Practices-Better Support of Weapon System Program 
Managers Needed to Improve Outcomes, GAO-06-110, 2006, 42. 

42 Ibid, 44. 

43 Ibid, 45. 
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changes to requirements due to several factors including; length of the program, 
emerging technologies, changes in mission or changes in funding. Frequently the need for 
a weapon no longer exists but the services do not want to cancel the program because 
extensive time and money has gone into the program, so instead of treating the program 
as a sunk cost and cancelling it the services develop a weapon system that is not needed. 
C. RISK MANAGEMENT 

Dobbins' and Donnelly's research concluded that risk management was a Critical 
Success Factor within DoD acquisition programs. Risk Management encompasses all 
aspects of risk within a program to include technological risk as well as funding and 
schedule risk. In 2002 Dobbins described the process of risk management: 

The fundamental notion is that we identify risk, we assess their probability 
of occurrence, and we assess the consequence of occurrence. Then we put 
a risk management plan in place that is designed to alleviate the impact of 
the serious event. Every risk is necessarily a future event, and only when 
the risk event actually happens is the risk transformed into a problem. The 
better we are at identifying risks and understanding the underlying basis of 
our risks, the better we can manage the risks.44 

The DoD Guide to Risk Management defines risk and risk management in the following 
manner, "Risk is a measure of the potential inability to achieve overall program 
objectives within defined cost, schedule, and technical constraints and has two 
components: (1) the probability/likelihood of failing to achieve a particular outcome, and 
(2) the consequences/impacts of failing to achieve that outcome." It goes on to define the 
process of risk management as: 

Risk management is the act or practice of dealing with risk. It includes 
planning for risk, assessing (identifying and analyzing) risk areas, 
developing risk-handling options, monitoring risks to determine how risks 
have changed, and documenting the overall risk management program.^s 


James Dobbins, Critical Success Factor Analysis for DoD Risk Management, Program Management 
Journal, May 2002, 40. 

45 Risk Management Guide for DoD Acquisitions (5th ed.), 2003, 7. 
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Dobbins and Donnelly describe risk management in broad terms and use the 
overarching concept of risk management as one of their Critical Success Factors. As 
Dobbins and Donnelly use the term "risk management", all aspects of a program can be 
considered part of the risk management process, from funding stability which causes 
budget risk to technical immaturity which is one aspect of technical risk. Other research 
has also identified risk management as critical to the success of a program but many 
researchers have defined the elements of risk management.46 For example, Delano uses 
several categories to describe risk including "Program's acquisition strategy", and "High 
degree of technical difficulty". While neither of these categories mentions risk 
management, both are strategies to reduce budget, schedule and technical risk and are 
part of an overarching risk management plan. It is clear that Delano also sees risk 
management as critical to the success of DoD acquisition programs. 

The GAO has also found issues with technical risk with many of the weapon 
systems being developed by the DoD. The GAO notes that many weapons are developed 
with immature technology that is key to the system. This usually creates unacceptable 
risk for most programs. When technologies are not mature and the system must have that 
capability to perform properly, the program runs the risk of going over budget, or over 
schedule, or both because the key technology did not mature fast enough. The GAO 
commented: 

Our most recent annual assessment of major weapon systems programs, 
showed that only 15 percent of the programs we reviewed began 
development having demonstrated that all of their technologies were 
mature. More often than not, programs had to worry about maturing 
technologies well into system development, when they should have 
focused on maturing system design and preparing for production. These 
assessments also show that programs that started development with mature 
technologies experienced lower development and unit cost increases than 
those programs that started with immature technologies.^^ 


46 Kenneth J. Delano, Identifying Factors That Contribute to Program Success, Acquisition Review 
Quarterly Winter-1998, 1998, 39. 

4^ Government Accountability Office, Best Practices-Better Support of Weapon System Program 
Managers Needed to Improve Outcomes, GAO-06-110, 2006, 41. 
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As the GAO points out, immature technologies often exacerbate budget and schedule 
problems because it takes longer and costs more than if the technology was already at a 
readiness level that was useful to the program. This is a theme that is seen in repeated in 
several GAO reports, including GAO-06-585T, where the GAO found: 

...DoD commits to its programs before it obtains assurance that the 
capabilities it is pursuing can be achieved within the available resources 
and time constraints. Funding processes encourage this approach, since 
acquisition programs attract more dollars than efforts concentrating solely 
on proving out technologies. Nevertheless, when DoD chooses to extend 
technology invention into acquisition, programs experience technical 
problems that have reverberating effects and require large amounts of time 
and money to fix.^s 

From the data that was presented in Chapter three it is clear that risk management 
is a Critical Success Factor and researchers, professionals and the GAO agree that risk 
management must be carefully addressed in order to develop successful weapon system 
acquisition programs. This is difficult to do within the DoD because of the need to 
develop weapons with greater capability and survivability. Most program managers 
would want to include any technology if they thought it would save a service-members 
life in a combat environment. The trade-off between cost and performance is a tough 
choice to make when it involves someone life. While trade-offs must be made it clear that 
program managers must manage risk in order to develop a successful program. 

D. TECHNICALLY COMPETENT PROGRAM OFFICE STAFF 

Dobbins and Donnelly identified a technically competent program staff as the 
fourth most identified Critical Success Factor mentioned by both Embedded Systems 
Program Managers and the Automated Information Systems Program Managers. As was 
pointed out in Chapter III, program managers specifically mentioned expertise in systems 
engineering as part of a technically competent staff. Further research by others identified 


Government Accountability Office, Defense Acquisition-Actions Needed to get Better Results on 
Weapons Systems Investments, GAO-06-585T, 2006, 6. 
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the same staffing/personnel problems, including the program managers interviews and 
literature review that was conducted by Delano, and interviews and case studies 
completed by the GAO. ^9 

Delano's research also identified a technically competent staff as a key factor 
within acquisition programs. His reference to "type and quality of people associated with 
the program" and "resources: people" are comparable to Dobbins' and Donnelly's 
"technically competent staff." Delano's literature review also found that attribute "quality 
people" was considered critical and was mentioned in 47 percent of the literature that he 
researched. 

Program managers interviewed by the GAO, also identified staffing problems as a 
major obstacle to successful programs. Specifically, the GAO stated, "...program 
managers comments and interviews with program executive officers pointed to critical 
shortages for staff that support them-including program management, systems 
engineering, cost estimating and software development. "50 

Further research conducted by the GAO and the DoD have also revealed that 
technically competent and professional staff are a key component of the acquisition 
process. The GAO has done several studies on the Government workforce and one of 
these reports was presented in Chapter Three as data regarding the importance of a 
technically competent workforce within the DoD and specifically, competent program 
staff within DoD acquisition programs.5i The DoD also has identified the need to educate 
and train the acquisition workforce to ensure they are technically competent regarding the 
acquisition process. The Defense Acquisition Workforce Improvement Act certification 
was the system that answered the question, "how does the DoD ensure they have a 
technically competent and qualified workforce?" The DAU website states, "The DAU 


^9 Kenneth J. Delano, Identifying Factors That Contribute to Program Success, Acquisition Review 
Quarterly Winter-1998, 1998, 38. 

50 Government Accountability Office, Best Practices-Better Support of Weapon System Program 
Managers Needed to Improve Outcomes, GAO-06-110, 2006, 44-45. 

51 Government Accountability Office, Acquisition Workforce-Status of Agency Effort to Address 
Future Need, GAO-03-55, 2003, 17. 
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provides a highly structured sequence of courses needed to meet the mandatory and 
desired training standards established in DoD 5000.52-M, “Career Development Program 

for Acquisition Personnel.”^2 

The DoD continues to face challenges to its workforce. The DoD must provide 
adequately trained and educated staff to support the complex weapon system programs 
that are developed and produced by the DoD. Over the past decade the DoD has reduced 
the acquisition workforce by half, while still trying to maintain qualified personnel to 
assist program managers, the loss of many experienced people has reduced the overall 
effectiveness of the acquisition system.53 Furthermore, it is imperative that acquisition 
professionals receive both education and training. DAWIA certification is a excellent 
training tool but must be coupled with education in a specific discipline such as systems 
engineering, program management, operational research etc. While the challenge to train 
and educate staff is critical it is also critical to have the right people in sufficient numbers 
at the right place to effectively staff a program office. It is clear from the data presented 
in Chapter III that program managers and other program professionals as well as the 
GAO have identified a technically competent staff as a critical component of a DoD 
programs. 

E. SUMMARY 

There have been a number of documents written on Critical Success Factors and 
their application to the DoD acquisition process and there has also been work on issues 
that continue to plague DoD acquisition programs. While much of the research does not 
acknowledge the existence of Critical Success Factors as it applies to DoD acquisition 
many authors are attempting to find a set of criteria that can be identified in most 
programs that will be predictive in nature, and allow program managers and program 
executives to use the criteria to assess the probability of a positive outcome for the 
program. The analysis that has been done in this chapter shows that there is a correlation 
between the Critical Success Factors researched and the success of the program. 


32 http://www.dau.mil/plm/plm.asp. DAU Website, 2006. 

33 Government Accountability Office, Acquisition Workforce-Status of Agency Efforts to Address 
Future Needs, GAO-03-55, 2002, 17. 
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The research of Dobbins and Donnelly, Delano, The Rand Corp and The 
Government Accountability Office make it clear that the DoD can benefit from a 
systematic approach to identifying factors that lead to the successful completion of 
programs. While not all programs will benefit from a set of criteria, it is imperative that 
Critical Success Factors are identified and measured for most programs in order to have a 
repeatable process for predicting the success of future DoD acquisition programs. Critical 
Success Factors will help to eliminate some of the variability in the acquisition process 
and provide a more stable acquisition environment. 
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V. RECOMMENDATIONS AND CONCLUSIONS 


A. CONCLUSIONS 

The DoD is continuing to buy more complex and sophisticated weapons that cost 
more money to develop and produce, and has lead to a higher per unit price tag and the 
purchase of smaller, inefficient quantities. The DoD continues to face Congressional 
scrutiny of its acquisition programs because of recurring problems with weapon system 
acquisition.54 The DoD must develop an approach to weapon system development and 
acquisition that produces superior weapons while maintaining the defense industrial base 
and keeping the trust and confidence of the U.S. taxpayer. The use of Critical Success 
Factors can help toward accomplishing this goal. 

From the data that was gathered regarding Critical Success Factors it is clear that 
while program management by definition is a one time undertaking that produces a 
unique and specific product, the systems engineering process that program management 
is based on has enough structure to allow the use of Critical Success Factors to identify 
possible problems within a program. 

The goal of this research was to answer the question: 

• What are the Critical Success Factors (CSF) that influence the successful 
acquisition of DOD weapon system programs? 

From the data that was collected and the analysis that was done, it is clear that 
there are several Critical Success Factors that can help determine the success of DoD 
weapon system acquisition programs. These four factors are: 

• Stable and Adequate Funding 

• Clearly Defined and Stable Requirements 

• Risk Management 

• Technically Competent Program Staff 

While there are many factors that can affect a program positively it is clear that these four 
factors listed above are seen by program managers, acquisition professionals and the 

54 www.govexec.eom/dailvfed/0705.htm . McCain Amendment Would Increase Defense Acquisition 
Oversight, 06 December 2006. 
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Government Aeeountability Offiee as the key elements to a sueeessful aequisition 
program. Dobbins' and Donnelly's researeh identified 18 Critieal Sueeess Faetors related 
to DoD acquisition programs. ^5 Many were not addressed in any detail by other authors 
or the GAO. The four Critical Success Factors that are mentioned above have been 
identified extensively in books, journals and government documents detailed in Chapter 
III and there is extensive information on why these four CSFs are key to the success of 
DoD acquisition programs. There was insufficient data to conduct analysis on the other 
14 Critical Success Factors identified by Dobbins and Donnelly. 

To help answer the primary research question the following secondary questions 
were also asked, and answered: 

• How do we define the success of an acquisition program? 

As stated in Chapter III, the way an organization defines success often depends on 
where that organization is in the acquisition process. For example, the user's definition of 
success does not necessarily coincide with the contractor's definition and vice versa.56 It 
is also apparent that each organization uses its own definition of success and from the 
research done there has not been one definition of success that is accepted by all the 
stakeholders. 

B. RECOMMENDATIONS 

• The DoD must recognize that Critical Success Factors exist and are the 
primary reason why programs succeed. The DoD must formally identify 
the Critical Success Factors that apply to DoD acquisition programs and 
implement a program to analyze and measure these factors in order to 
improve the success of future programs. 

• The DoD must institutionalize the lessons on Critical Success Factors and 
ensure that future program managers understand the process and can 


James H. Dobbins. & Richard G. Donnelly, Summary Research Report on Critical Success Factors 
in Federal Government Program Management, Acquisition Review Quarterly Winter-98, 1998, 62. 

56 Defense Acquisition University, Introduction to Defense Acquisition Management, 1999, 7. 
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successfully implement the program. This requires "buy in" from senior 
acquisition executives that support and understand the importance of 
Critical Success Factors. 

• The formal education and training of program managers through the 
Defense Acquisition University must include the process of identifying 
Critical Success Factors and implementing the measures to ensure 
successful programs. It is not enough to understand what critical factors 
are, there must be a systematic process for identifying and measuring 
factors that are key to the success of the acquisition program. 

• The DoD, the Congress, the user and the acquisition community must 
agree to the definition of what a successful program is. Until the major 
stakeholders can agree to what success means and share a common vision 
then future programs will continue to have major problems. 

C. AREAS OF FURTHER RESEARCH 

There are two areas that further research should explore. First, it is clear that more 
research must be done in the area of Critical Success Factors. There must be a concerted 
effort to identify and measure the effects of Critical Success Factors on DoD acquisition 
programs. There is not enough data to validate the next 14 Critical Success Factors that 
were identified by Dobbins and Donnelly, so more research need to be done in this 
particular area. Further research should focus on documenting the effect that the 14 
Critical Success Factors have had on past programs and possible predictions on future 
programs. 

The second area that needs to be explored is the relationship between internal and 
external factors and the impacts that the two have on programs. This research focused on 
internal factors that, for the most part, could be controlled within DoD, future research 
needs to focus on the relationship between outside factors and program success, to 
include the budget process and how it creates instability within programs. 

The DoD acquisition process produces the most technically advanced and lethal 
weapon systems in the world, and our military is unmatched in its capability to project 
lethal forces around the world. This reality must be tempered with the understanding that 
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defense budgets are coming under greater scrutiny and control and as our weapon 
systems become more complex we buy fewer of them. The DoD must become better at 
producing weapons in a cost and time constrained environment. If the DoD can adopt a 
basic guideline for using Critical Success Factors to guide a program to completion it will 
streamline the process and create a more efficient process for weapon systems 
acquisition. 
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